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Preface 


The  National  Aeronautics  and  Space  Administration  (NASA)  is  required  to 
meet  the  requirements  of  the  President's  September  1993  Executive  Order  12862, 
Setting  Customer  Service  Standards.  A  Headquarters  steering  group  is  developing 
a  plan  to  meet  those  requirements.  That  NASA  agency  plan  is  due  to  the  Presi¬ 
dent  by  8  September  1994. 

The  Logistics  Management  Institute  (LMI)  has  been  engaged  to  provide  a 
common  approach  for  planning,  conducting,  and  analyzing  customer  satisfaction 
surveys.  LMI  has  also  been  asked  to  develop  a  performance  measurement 
framework  for  consistently  undertaking  and  gauging  improvement  initiatives. 
In  this  way,  NASA  will  be  in  a  position  to  consolidate  individual  customer  serv¬ 
ice  plans  into  an  agency  plan. 

The  Logistics  Management  Institute  will  be  available  for  advice  and  guid¬ 
ance  in  accomplishing  these  improvement  initiatives.  The  authors  can  be 
reached  at  telephone  number  (703)  917-9800  for  such  assistance. 
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Logistics  Management  Institute 


An  Approach  for  Meeting  Customer  Standards 
Under  Executive  Order  12862 


Summary 


Executive  Order  12862,  Setting  Customer  Service  Standards,  requires  that 
NASA  take  customer  satisfaction  surveys  to  obtain  customer  feedback  for  im¬ 
proving  its  products  and  services.  It  also  requires  NASA  to  search  for  best  in¬ 
dustry  models  and  incorporate  them  into  current  practices  as  appropriate. 
Executive  Order  12862,  however,  does  not  provide  much  in  the  way  of  guide¬ 
lines  for  its  implementation. 

This  report  provides  a  comprehensive  set  of  guidelines  for  complying  with 
the  provisions  of  Executive  Order  12862.  The  guidance  is  presented  as  follows. 

Chapter  1  introduces  the  requirements  of  Executive  Order  12862  and  shows 
the  relationship  of  those  requirements  to  other  NASA  improvement  efforts. 

Chapter  2  outlines  NASA's  Strategic  Enterprise  Units  and  their  customers. 
It  also  defines  an  appropriate  level  of  process  aggregation  for  the  Strategic  Enter¬ 
prise  Units  to  design  customer  satisfaction  surveys  that  can  lead  directly  to  per¬ 
formance  improvements  for  NASA. 

Chapter  3  delineates  a  quality  measurement  framework  to  guide  the  focus  of 
customer  satisfaction  surveys.  It  also  provides  practical  advice  on  planning,  con¬ 
ducting,  and  analyzing  the  customer  surveys  as  well  as  a  basis  for  prioritizing 
performance  improvement  initiatives. 

Chapter  4  offers  a  total  performance  measurement  framework— combining 
cost,  output,  and  quality— to  further  guide  the  accomplishment  of  benchmark¬ 
ing,  continual  improvement,  and  implementation  of  the  Government  Perform¬ 
ance  and  Results  Act  of  1993. 
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Chapter  1 

Introduction 


During  the  spring  of  1992,  the  National  Aeronautics  and  Space  Administra¬ 
tion  (NASA)  embarked  on  corporate  approach  of  continual  improvement  to  en¬ 
hance  customer  satisfaction  with  its  products  and  services.  Figure  1  illustrates 
an  improvement  cycle. 


Figure  1. 

LMI's  Concept  of  an  Improvement  Cycle 


The  five  elements  of  NASA's  improvement  cycle  are  as  follows: 

1.  Customers  judge  the  results  (through  the  use  of  advisory  groups  and  formal 
surveys). 

2.  Process  definition  provides  the  understanding  for  addressing  customer  con¬ 
cerns. 

3.  Management  and  employee  groups  (the  stakeholders)  work  in  teams  to  im¬ 
prove  processes. 

4.  Leadership  gives  focus,  alignment,  and  direction  to  team  efforts. 
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5.  Tools  and  techniques  are  used  to  bring  about  process  improvement;  they  in¬ 
clude  cross-functional  problem  solving,  controlled  testing,  and  performance 
measurement. 

Like  other  government  agencies,  NASA  is  required  to  pursue  two  related 
improvement  initiatives.  First,  in  September  1993,  the  President  issued  Executive 
Order  12862,  Setting  Customer  Service  Standards,  to  help  implement  the  recom¬ 
mendations  of  the  National  Performance  Review.  Executive  Order  12862,  as 
with  NASA's  continual  improvement  effort,  focuses  on  reducing  costs  and  in¬ 
creasing  product/service  quality  to  improve  customer  satisfaction.  However, 
the  Executive  order  places  special  emphasis  on  taking  formal  surveys  to  obtain 
customer  feedback  and  on  benchmarking  related  industry  practices  to  improve 
government  processes.  Government  agencies  are  to  begin  those  activities  in 
FY94. 

Second,  the  Government  Performance  and  Results  Act  of  1993  requires 
NASA  not  only  to  improve  costs  and  enhance  quality,  but  also  to  focus  on  out¬ 
puts  and  performance  budgeting.  It  further  requires  pilot  tests  to  develop  strate¬ 
gic  plans,  to  construct  operating  plans,  and  to  formulate  performance  budgets  (in 
terms  of  output,  quality,  and  cost).  The  pilot  tests  are  to  be  completed  by  Sep¬ 
tember  1997,  with  a  full  performance  budgeting  system  in  operation  by  the  year 
2000. 

We  believe  that  all  of  these  efforts— continual  improvement.  Executive  Or¬ 
der  12862,  and  the  Government  Performance  and  Results  Act  of  1993— should  be 
thought  of  as  integral  to  an  overall  performance  improvement  system.  Certainly, 
an  improvement  cycle  is  needed  to  make  the  other  two  efforts  effective.  Also,  a 
cohesive  performance  measurement  system  provides  the  capability  to  focus  on 
all  aspects  of  performance  (cost,  quality,  and  output),  coordinate  improvement 
efforts,  and  gauge  the  results. 


Executive  Order  12862 

Executive  Order  12862  promotes  government  service  to  the  public  that 
matches  or  exceeds  the  best  service  available  in  the  private  sector.  It  focuses  on 
establishing  and  implementing  customer  standards  to  help  guide  government 
operations.  A  customer  is  defined  as  "an  individual  or  entity  who  is  directly 
served  by  a  department  or  agency."  The  Executive  order  is  shown  in  its  entirety 
in  Appendix  A. 

Specifically,  the  Executive  order  requires  that  NASA  take  customer  satisfac¬ 
tion  surveys  to  obtain  customer  feedback  for  improving  its  products  and  serv¬ 
ices.  It  also  requires  NASA  to  search  for  best  industry  and  public-sector  methods 
and  incorporate  them  into  current  practices  as  appropriate.  The  Executive  order, 
however,  does  not  provide  much  in  the  way  of  guidelines  for  its  implementation. 
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The  National  Aeronautics  and  Space  Administration  needs  a  comprehensive 
set  of  guidelines  for  complying  with  the  general  provisions  of  Executive  Order 
12862.  This  guidance  answers  the  following  questions: 


Who  in  the  NASA  organization  should  be  taking  customer  satisfaction  sur¬ 
veys  and  searching  for  best  industry  methods? 

What  should  NASA  measure  to  obtain  meaningful  customer  feedback  and 
relevant  industry  information  for  improving  performance? 

How  should  NASA  approach  customer  satisfaction  surveys? 

When  should  NASA  undertake  customer  satisfaction  surveys,  and  search 
for  best  industry  or  public-sector  practices? 


Organization  of  Report 

This  guidance  presents  a  strategy  for  NASA  to  meet  the  requirements  of  Ex¬ 
ecutive  Order  12862,  to  further  its  continual  improvement  efforts,  and  to  prepare 
for  the  challenges  of  the  Government  Performance  and  Results  Act  of  1993. 

Chapter  2  discusses  NASA's  customers  and  organizational  structure  and  the 
concept  of  a  process  for  diagnosing  problems  and  making  process  improve¬ 
ments. 

Chapter  3  focuses  on  meeting  the  requirements  of  Executive  Order  12862.  It 
defines  a  measurement  framework  for  assessing  customer  satisfaction  and  out¬ 
lines  the  basic  steps  for  planning,  conducting,  and  analyzing  customer  satisfac¬ 
tion  surveys. 

Chapter  4  presents  a  more  complete  performance  measurement  system.  It 
contains  the  measurement  framework  for  assessing  customer  satisfaction  dis¬ 
cussed  in  Chapter  3  and  other  performance  measures  that  are  needed  to  meet  the 
benchmarking  requirements  of  Executive  Order  12862,  continual  improvement 
efforts,  and  the  Government  Performance  and  Results  Act  of  1993. 
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Chapter  2 

Definition  of  a  Process 


Everything  NASA  does  involves  a  process.  Every  process  includes  one  or 
more  inputs  (facilities,  management,  and  research  staff),  a  variety  of  transforma¬ 
tions,  and  various  outputs  and  levels  of  quality.  Figure  2  illustrates  the  most 
simple  process,  with  customer  and  supplier  feedback. 
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Figure  2. 

A  Simple  Process 


Customers 


The  National  Aeronautics  and  Space  Administration  is  currently  undertak¬ 
ing  an  input/  output  analysis  to  fully  identify  its  customers,  products  and  serv¬ 
ices,  inputs,  suppliers,  and  associated  processes.  In  broad  terms,  that  analysis 
has  shown  that  NASA's  external  customers  include  the  science  and  education 
communities,  the  aerospace  industry,  the  aeronautics  and  aviation  industries, 
and  other  government  agencies,  as  well  as  the  American  public,  the  media,  the 
Administration,  and  Congress.  Five  Strategic  Enterprise  Units  (SEUs)  have  been 
defined  to  focus  on  specific  strategic  objectives  and  external  customers. 

The  National  Aeronautics  and  Space  Administration's  five  SEUs  are  Mission 
to  Planet  Earth,  Aeronautics,  Human  Exploration  and  Development  of  Space, 
Scientific  Research,  and  Space  Technology.  The  SEUs  differ  in  their  customer  fo¬ 
cuses.  For  example,  the  Aeronautics  Enterprise  serves  the  aeronautics  industry 
and  other  U.S.  government  agencies,  while  the  Space  Technology  Enterprise 
serves  the  aerospace  industry.  However,  some  SEUs  serve  some  of  the  same  ex¬ 
ternal  customers.  For  example,  although  many  of  the  external  customers  of  the 
Mission  to  Planet  Earth  Enterprise  and  the  Scientific  Research  Enterprise  differ 
substantially,  both  SEUs  serve  the  science  community  in  various  ways.  Also, 
some  of  the  customers  are  general  rather  than  specific  —  such  as  customers  of 
public  affairs,  publications,  educational  programs,  and  procurement.  The 
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activities  serving  them  are  performed  across  SEUs  and  at  Headquarters;  we  refer 
to  them  as  crosscutting  activities. 

The  National  Aeronautics  and  Space  Administration  Headquarters  has  indi¬ 
cated  that  each  SEU  and  the  crosscutting  area  will  be  responsible  for  developing 
a  customer  service  plan  to  meet  the  requirements  of  Executive  Order  12862.  Each 
SEU  should  plan  and  conduct  at  least  one  pilot  customer  survey  for  a  non¬ 
crosscutting  activity  during  the  next  6  months.  The  Associate  Administrator  for 
continual  improvement,  in  cooperation  with  the  SEUs,  will  focus  on  a  crosscut¬ 
ting  customer  survey.  We  now  address  the  appropriate  process  aggregation 
level  for  SEUs  to  undertake  customer  satisfaction  surveys  and  other  performance 
initiatives. 


NASA  Processes 

The  National  Aeronautics  and  Space  Administration  needs  to  define  the  ap¬ 
propriate  operating  level  in  its  organization  for  defining  its  processes  and  for 
meeting  the  requirements  of  Executive  Order  12862  and  other  performance  im¬ 
provement  initiatives.  We  believe  that  NASA  should  not  define  its  processes  too 
broadly  or  too  narrowly  to  effectively  undertake  performance  improvement. 

In  principle,  we  should  consider  at  least  six  levels  of  aggregation  for  defin¬ 
ing  NASA's  processes.  First,  NASA  could  consider  the  products  and  services  it 
produces  with  other  agencies,  such  as  the  Department  of  Defense  or  the  Depart¬ 
ment  of  Commerce,  as  a  process.  Second,  NASA  could  consider  its  agency  as  a 
whole  as  a  process.  Third,  at  a  somewhat  more  detailed  level,  NASA  could  con¬ 
sider  each  of  its  five  SEUs  as  a  separate  process.  Fourth,  NASA  could  consider 
its  12  current  budget  programs  (e.g..  Launch  Services  or  Physics  and  Astronomy) 
as  individual  processes.  Fifth,  at  a  still  more  disaggregated  level,  NASA  could 
define  separate  processes  for  the  specific  products/services  it  provides  at  each  of 
its  nine  research  centers  or  at  its  Headquarters.  (As  an  example,  one  process 
could  consist  of  all  wind  tunnels  at  Ames  Research  Center.)  Sixth,  at  an  even 
more  detailed  level,  NASA  could  define  separate  processes  for  each  of  its  func¬ 
tional  areas,  such  as  maintenance  and  repair  of  buildings,  facilities,  and  struc¬ 
tures. 

The  National  Aeronautics  and  Space  Administration  needs  some  criteria  to 
select  the  most  appropriate  definition  of  process  aggregation  for  undertaking 
performance  improvement  initiatives.  We  believe  that  four  criteria  have  applica¬ 
tion  to  NASA.  A  process  should  do  the  following: 

1.  Encompass  products/ services  that  are  produced  jointly  through  the  use  of 
common  staff,  equipment,  facilities,  and  technology.  In  economics,  this  cri¬ 
terion  would  be  referred  to  as  the  production  function. 

2.  Come  under  the  control  of  a  manager  who  is  directly  responsible  for 
strengthening  operations  and  improving  customer  satisfaction  related  to 
certain  products/  services.  For  NASA,  this  criterion  means  that  each  process 
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needs  to  be  defined  in  terms  of  the  relevant  research  center  or  headquarters 
that  oversees  its  operation. 


3.  Serve  as  a  diagnostic  tool  for  taking  action  if  specific  performance  improve¬ 
ments  are  warranted  by  customer  feedback.  This  criterion  means  that  the 
process  is  keyed  to  the  products/ services  provided  to  specific  customers, 
and  the  process  is  detailed  enough  to  pinpoint  actionable  items. 

4.  Point  to  areas  in  which  industry  and  public-sector  best  practices  might  be 
useful  for  making  improvements.  NASA  should  be  able  to  identify  specific 
functions  (or  a  group  of  functions)  from  its  processes,  search  for  best  models 
that  relate  to  those  functions,  and  incorporate  them  into  its  processes.  In  the 
continual  improvement  literature,  this  practice  is  referred  to  as  benchmark¬ 
ing.1 

We  now  apply  these  criteria  to  select  the  most  appropriate  level  of  aggrega¬ 
tion  for  defining  NASA  processes. 

Table  1  is  a  matrix  of  the  four  alternative  process  aggregation  levels  (the 
rows)  and  the  four  selection  criteria  (the  columns).  We  indicate  a  "yes"  in  the 
cells  when  the  process  aggregation  levels  satisfy  the  specific  criteria  for  process 
aggregation  and  "no"  when  they  do  not. 


Table  1. 

Selection  of  Process  Definition 


Process  aggregation  level 

Criteria  for  selection 

Production 

function 

Direct  man¬ 
ager  control 

Diagnostics  for  im¬ 
proving  customer 
satisfaction 

Industry  bench¬ 
marking 

Govemmentwide 

No 

No 

No 

No 

NASA 

No 

No 

No 

No 

Strategic  enterprise  unit 

Yes 

No 

No 

No 

Budget  program 

Yes 

No 

No 

No 

Products/services  at  re¬ 

Yes 

Yes 

Yes 

Yes 

search  centers  or  Head¬ 

quarters 

Function 

Yes 

Yes 

No 

Yes 

At  one  extreme,  the  matrix  shows  that  defining  detailed  processes  at  the 
Govemmentwide,  NASA,  or  strategic  enterprise  unit  level  is  too  broad  for  un¬ 
dertaking  performance  improvements.  At  the  other  extreme,  defining  processes 
at  the  functional  level  is  too  narrow  to  permit  diagnosis  of  all  potential  customer 


1  See  Robert  C.  Camp,  Benchmarking:  The  Search  for  Industry  Best  Practices  that  Lead  to 
Superior  Performance,  White  Plains,  New  York:  Quality  Resources,  1989,  for  an  in-depth 
treatment  of  benchmarking. 
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concerns  regarding  a  specific  product  or  service.  As  Table  1  shows,  all  four  cri¬ 
teria  are  satisfied  only  for  the  products/  services  at  each  of  NASA's  research  cen¬ 
ters  or  at  Headquarters.  It  is  at  this  level  of  process  aggregation  that 
performance  improvement  initiatives  align  with  the  managers  who  directly  diag¬ 
nose  performance  problems,  consider  the  best  practices  of  industry,  and  make 
process  improvements. 

We  conclude  that  NASA  should  define  its  processes  along  product  or  service 
lines  at  each  delivery  level.  For  example,  the  Ames,  Lewis,  and  Langley  Re¬ 
search  Centers  all  operate  and  manage  wind  tunnels  for  both  NASA  and  indus¬ 
try  customers.  Nonetheless,  a  separate  wind-tunnel  process  should  be  defined 
for  each  of  the  three  research  centers.  If  this  is  done,  the  three  separate  wind- 
tunnel  processes  would  reflect  the  differences  in  their  operations,  thereby  foster¬ 
ing  customer  feedback,  industry  benchmarking,  and  improvement  initiatives. 
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Chapter  3 

Executive  Order  12862 


Executive  Order  12862  requires  NASA  to  develop  customer  satisfaction  sur¬ 
veys  and  performance  benchmarking  in  order  to  improve  its  operations.  We  be¬ 
lieve  that  NASA  should  first  focus  on  taking  customer  satisfaction  surveys  (for  at 
least  a  year)  before  undertaking  benchmarking.  This  will  give  NASA  an  oppor¬ 
tunity  to  understand  its  customer  standards  well,  a  necessary  first  step  before 
considering  the  introduction  of  best  industry  standards. 

Also,  benchmarking  requires  not  only  a  focus  on  quality— as  is  the  case  for 
customer  satisfaction  surveys— but  also  on  cost  and  output  performance.  NASA 
will  have  to  develop  cost  and  output  measures  before  it  can  properly  undertake 
industry  benchmarking.  Chapter  4  provides  the  broader  measurement  system 
that  NASA  will  need  to  comply  with  the  benchmarking  requirements  of  the  Ex¬ 
ecutive  order  (and  with  continual  improvement  and  the  Government  Perform¬ 
ance  and  Results  Act  of  1993). 

Presented  below  is  the  quality  measurement  approach  needed  for  meeting 
the  customer  survey  requirements  of  the  Executive  order.  It  is  followed  by 
guidance  for  planning,  conducting,  and  analyzing  customer  surveys. 


Quality  Measurement 

Executive  Order  12862  basically  requires  that  the  customer  survey  effort  pro¬ 
vide  data  for  determining  the  status  of  quality  in  NASA  products/  services,  for 
prioritizing  NASA  improvement  initiatives,  and  for  gauging  overall  progress. 

The  Executive  order  requires  that  the  customer  survey  instrument  cover  the 
following  four  areas: 

♦  Customer  standards  for  each  quality  dimension  of  a  product/  service 

♦  Customer  satisfaction  levels  for  every  product/  service  quality  dimension 

♦  Customer  importance  attached  to  each  product/ service  quality  dimension 

♦  Customer  concern  with  product/ service  expenditures. 

Surveys  need  to  be  structured  so  that  the  respondents  register  their  feelings 
in  quantitative  terms.  Such  scoring  should  be  done  for  each  of  the  customer 
standards,  satisfaction  levels,  importance  levels,  and  expenditure  concerns.  The 
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resultant  data  then  can  be  combined  to  meet  Executive  Order  12862  require¬ 
ments. 

Equation  1  shows  how  to  combine  much  of  the  data  into  a  quality  index.  In 
accordance  with  our  definition  of  process,  the  quality  index  (QI)  represents  “n" 
quality  factors  of  a  product/  service  that  is  produced  by  a  research  center  or  by 
Headquarters  for  a  certain  set  of  customers.  Scores  on  actual  and  standard  levels 
are  compared  and  then  weighted  by  relative  customer  importance  scores.  (These 
scores  are  obtained  through  the  customer  survey  as  explained  later  in  this  chap¬ 
ter.) 


QI=w(  l)x 


actual  quality^  1) 
standard  quality^}) 


v  /  standard  quality (2) 


w(n)x 


actual  qualityiri) 
standard  quality(n)  * 


[Eq.  1] 


Equation  1  is  a  useful  tool  for  determining  the  quality  factors  most  in  need 
of  improvement.  As  an  example,  if  two  quality  factors  achieve  only  40  percent  of 
their  standard  levels,  then  the  one  with  the  greater  customer  weight  should  re¬ 
ceive  the  highest  priority  for  making  improvements.  In  contrast,  a  quality  factor 
with  an  80  percent  achievement  level  relative  to  its  standard  (and  a  relatively  low 
customer  weight)  should  receive  a  lower  priority  for  implementing  improve¬ 
ments.  Moreover,  these  quality  shortfalls  also  should  be  compared  with  cus¬ 
tomer  expenditure  concerns  for  the  product/  service.  Finally,  it  is  possible  that 
customer  expenditure  concerns  are  as  great  as  or  greater  than  any  particular 
quality  shortfall,  and  that  should  be  taken  into  account  before  prioritizing  NASA 
improvement  efforts.  See  Chapter  4  for  a  more  complete  discussion  of  customer 
expenditure  concerns  and  NASA  costs  for  producing  products/  services. 

The  quality  index  is  also  a  useful  tool  for  measuring  the  degree  to  which 
overall  quality  standards  are  met.  For  example,  a  70  percent  quality  index  score 
would  mean  that,  on  the  average,  the  process  is  achieving  70  percent  of  the  qual¬ 
ity  standards  desired  by  the  customers.  The  maximum  quality  index  score  is 
100  percent,  and  the  minimum  is  0  percent. 


Customer  Survey 


There  are  three  phases  for  undertaking  customer  surveys:  planning  for  the 
survey,  conducting  the  survey,  and  analyzing  its  results. 


Planning  for  the  Survey 

The  design  of  the  survey  includes  identifying  NASA's  customers,  structur¬ 
ing  the  questionnaire,  and  planning  for  sampling  the  customers.  All  of  these 
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considerations  are  important  for  ensuring  that  NASA  obtains  accurate  and  unbi¬ 
ased  feedback  from  its  customers. 

The  National  Aeronautics  and  Space  Administration  is  already  defining  its 
customer  base  in  precise  terms.  This  is  an  important  first  step  toward  meeting 
the  requirements  of  Executive  Order  12862. 

The  customer  questionnaire  should  be  developed  using  a  working  group  for 
each  process.  For  example,  a  working  group  could  be  formed  for  the  wind- 
tunnel  process  at  the  Ames  Research  Center.  This  group  could  include  the  man¬ 
ager  of  the  process,  selected  members  of  the  manager's  staff,  a  key  customer,  and 
a  survey  expert.  The  manager  and  staff  would  ensure  that  customer  feedback  is 
meaningful  for  making  improvements  to  their  process.  The  customer  participant 
would  increase  the  likelihood  that  important  product/ service  characteristics  are 
included  in  the  questionnaire,  while  the  survey  expert  would  help  the  group  to 
avoid  unclear  and  biased  wording  in  the  questionnaire. 

Survey  research  has  shown  that  a  mail  questionnaire  needs  to  follow  a  cer¬ 
tain  structure  to  obtain  an  acceptable  response  rate.  First,  it  should  include  a 
cover  letter  describing  the  survey's  importance.  The  cover  letter  should  include 
the  following  statements:  NASA  intends  to  take  periodic  surveys  to  determine 
how  well  it  is  meeting  the  needs  of  its  customers;  customer  feedback  will  help 
NASA  make  improvements  to  its  processes;  customer  cooperation  is  essential  for 
continually  improving  NASA's  products/ services;  and  NASA  guarantees  that 
individual  respondent  data  will  be  kept  confidential. 

Second,  the  survey  questions  should  be  written  in  sections:  one  section  on 
customer  satisfaction  levels,  another  on  customer  standards,  and  still  another  on 
the  importance  that  customers  attach  to  specific  product/ service  characteristics. 
The  questions  should  be  structured  so  that  respondents  can  express  their  feelings 
in  degrees  of  intensity.  Survey  research  further  shows  that  use  of  a  seven-point 
scale,  with  appropriate  descriptive  labels,  is  very  effective.  Using  the  example  of 
the  Ames  Research  Center  wind  tunnel,  a  survey  question  on  customer  satisfac¬ 
tion  levels  could  be  as  follows: 

How  satisfied  are  you  with  the  reliability  of  wind-tunnel  services 
provided  by  the  Ames  Research  Center? 

1  —  Not  satisfied. 

2  —  Slightly  satisfied. 

3  —  Somewhat  satisfied. 

4  —  Fairly  satisfied. 

5  —  Quite  satisfied. 

6  —  Very  satisfied. 

7  —  Extremely  satisfied. 

Each  section  should  conclude  with  a  question  that  elicits  customer  explana¬ 
tions  for  relatively  negative  answers  (such  as  1, 2,  or  3  in  the  above  sample  ques¬ 
tion)  or  general  comments. 
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After  the  questionnaire  sections,  the  survey  should  conclude  with  two  addi¬ 
tional  types  of  questions.  One  should  address  the  customer's  overall  rating  of 
NASA's  products/  services,  again  using  a  l-to-7  scoring  scale.  Such  a  question  is 
useful  for  checking  the  consistency  of  customer  responses  throughout  the  ques¬ 
tionnaire.  The  other  question  should  be  aimed  at  developing  a  business  profile 
of  the  customer.  That  information  should  be  useful  in  determining  whether  dif¬ 
ferent  types  of  customers  have  different  feelings  about  various  aspects  of 
NASA's  products/ services,  which  may  help  NASA  target  its  products/ services 
to  specific  customers. 

Before  conducting  the  survey,  each  working  group  should  pretest  the  ques¬ 
tionnaire  to  check  for  possible  bias  and  ensure  that  the  questions  are  clear  and  ef¬ 
fective.  The  pretesting  should  result  in  the  elimination  of  long,  awkwardly 
worded,  or  ambiguous  questions;  redundant  questions;  and  wording  that  may 
offend  or  sound  foolish  to  respondents.  Questionnaires  are  routinely  improved 
when  pretested. 

A  mail  survey,  by  itself,  is  often  not  sufficient  for  obtaining  an  adequate  sur¬ 
vey  response  rate.  The  Office  of  Management  and  Budget  requires  that  govern¬ 
ment  customer  satisfaction  surveys  strive  for  a  75  percent  response  rate.  NASA 
would  like  to  achieve  a  higher  response  rate.  Survey  research  shows  that  a  mail 
survey  often  achieves  only  a  40  percent  response  rate.  To  increase  that  response 
rate,  organizations  need  to  take  two  additional  actions  —  a  reminder  card  and  a 
follow-up  with  a  telephone  call  or  in-person  interview.  These  steps  —  a  mail 
survey,  followed  by  a  card  and  telephone  call  or  in-person  interview  —  are  re¬ 
ferred  to  as  the  "total  design  method"  in  survey  research  circles. 

Table  2  compares  the  various  survey  methods  and  illustrates  that  the  total 
design  method  often  achieves  acceptable  response  rates  at  reasonable  costs.2  A 
questionnaire  costs  about  $5.00  for  handling  and  mailing.  However,  because 
only  40  percent  are  returned,  the  cost  of  the  initial  and  follow-up  mail  question¬ 
naires  increases  to  approximately  $12.00  ($5/0.40)  in  order  to  obtain  satisfactory 
response  rates.  Telephone  and  in-person  interviews  are  more  expensive  than 
those  for  mail  questionnaires,  but  their  response  rates  are  considerably  higher 
than  those  for  mail  questionnaires.  The  total  design  method  combines  the  best 
of  all  survey  methods  by  balancing  costs  with  response  rate  considerations. 

The  sample  size  of  the  survey  affects  both  its  accuracy  and  cost.  When 
NASA  processes  involve  fewer  than  250  industry  and  internal  NASA  customers, 
we  suggest  that  NASA  survey  all  of  its  customers.  However,  when  the  customer 
base  exceeds  250  customers,  NASA  should  take  a  50  percent  sample.  Such  sam¬ 
pling  rate  procedures  balance  data  accuracy  considerations  against  survey  costs.3 


2See  D.A.  Dillman,  Mail  and  Telephone  Surveys:  The  Total  Design  Method,  New  York: 
John  Wiley  &  Sons,  1978,  for  an  in-depth  treatment  of  this  survey  method. 

3  A  random  sample  may  be  taken  when  the  customer  base  is  relatively  homogeneous. 
However,  when  the  customer  base  is  diverse,  the  method  of  stratified  sampling  is  more 
appropriate  for  obtaining  the  desired  data  accuracy.  See  George  W.  Snedecor 
and  G.  Cochran,  Statistical  Methods,  The  Iowa  State  University  Press,  1989,  for  a  practical 
discussion  of  sampling  techniques. 
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Table  2. 

Survey  Methods  and  Costs 


Average  cost  ($) 

Survey  method 

Response  rate 
(%) 

Per  attempted 
contact 

Per  successful 
contact 

Mail  questionnaire 

40 

5 

12 

Telephone  interview 

60 

20 

33 

In-person  interview 

70 

40 

57 

Total  design  method 

65 

17 

26 

Source:  Various  survey  research  documents. 


Notes:  Mail  questionnaire  costs  include  clerical  and  data  coding  functions.  In-person  interviews  do  not 
include  travel/lodging  costs.  Total  design  method  assumes  a  mixture  of  mail  questionnaires,  telephone  in¬ 
terviews,  and  in-person  interviews.  If  NASA  has  additional  internal  costs,  they  should  be  calculated  and 
considered. 


Conducting  the  Survey 

The  National  Aeronautics  and  Space  Administration  should  take  a  an  an¬ 
nual  survey  to  periodically  review  performance  progress  and  continuously  ob¬ 
tain  customer  feedback.  It  should  also  use  a  100  percent  sampling  rate  when  the 
customer  base  is  under  250  and  50  percent  when  more  than  250. 

The  National  Aeronautics  and  Space  Administration  should  pretest  every 
survey  instrument.  We  recommend  that  the  working  group  test  the  survey  in¬ 
strument  by  a  combination  of  telephone  interviews  and  in-person  interviews.  A 
5  percent  pretest  sample  should  be  appropriate.  A  useful  procedure  for  pretest¬ 
ing  is  to  ask  respondents  a  question  such  as,  "Why  did  you  give  such  a  rating  in 
answer  to  this  question?"  The  answers  may  indicate  that  the  respondents  are  in¬ 
terpreting  the  questions  differently  from  the  way  the  working  group  had  in¬ 
tended. 

After  the  pretest,  an  additional  6-week  period  should  be  set  aside  for  collect¬ 
ing  questionnaire  responses.  The  questionnaires  should  be  mailed  out  with  a 
stamped,  self-addressed  return  envelope.  After  the  second  week,  NASA  should 
mail  a  card  thanking  respondents  who  have  completed  the  questionnaire  and  re¬ 
minding  others  that  it  is  looking  forward  to  receiving  their  responses.  After  the 
fourth  week,  NASA  should  conduct  any  follow-up  telephone  interviews  and  per¬ 
sonal  interviews  necessary  to  achieve  high  response  rates. 

Expert  advice  is  especially  important  in  conducting  the  survey.  The  results 
will  be  accurate  only  if  based  upon  a  reasonable  cross  section  of  the  population. 
Further,  customer  interviews  — either  by  telephone  or  in  person— could  intro¬ 
duce  bias  into  the  survey  data  if  not  properly  conducted.  NASA  may  want  to 
contract  for  those  services. 
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Analyzing  the  Survey  Results 

Before  analyzing  the  data,  NASA  should  code  the  data  for  entering  into  a 
computer.4  It  should  make  plans  to  transfer  the  survey  data  into  the  computer. 
Entries  should  be  double-checked  to  ensure  accuracy. 

The  amount  of  information  yielded  by  customer  surveys  can  be  consider¬ 
able.  In  general,  analysis  of  survey  data  should  include  examination  of  the  re¬ 
sponses  to  every  question  for  each  category  of  respondent.  Such  information 
indicates  which  products/services  are  meeting  customer  standards  for  different 
categories  of  NASA  customers.  Also,  examination  of  the  responses  to  open- 
ended  comment  questions,  such  as  why  the  respondents  gave  negative  answers 
to  some  questions,  may  suggest  process  improvements.  Finally,  more  sophisti¬ 
cated  statistical  approaches  may  be  used  for  determining  the  relationship  be¬ 
tween  the  responses  to  various  questions,  thereby  improving  understanding  of 
customer  thinking. 

The  requirements  of  Executive  Order  12862  dictate  that  the  analysis  of  the 
survey  specifically  include  at  least  the  following: 

♦  A  determination  of  customer  standards  for  every  quality  dimension  of 
NASA's  products/  services.  These  standards  should  be  summarized  from 
the  quantitative  answers  provided  by  the  survey  respondents. 

♦  An  analysis  of  the  extent  to  which  customers  are  satisfied  with  the  quality  of 
NASA's  products/services.  Customer  satisfaction  scores  can  be  aggregated 
from  the  individual  customer  scores  provided  by  the  respondents. 

♦  An  evaluation  of  quality  gaps  between  NASA's  products/services  and  the 
customer's  expectations.  A  summary  of  the  delivery-expectation  quality 
gaps  can  be  obtained  from  the  quantitative  scores  provided  by  the  survey  re¬ 
spondents. 

♦  A  summary  of  the  relative  importance  that  customers  attach  to  each  quality 
dimension  of  NASA's  products/services.  Relative  importance  can  be  de¬ 
rived  by  comparing  the  customer's  importance  scores  on  the  quality  factors 
with  the  total  of  all  customer  quality-importance  scores. 

♦  An  assessment  of  NASA's  overall  quality  performance.  Such  an  assessment 
should  be  based  upon  the  construction  of  the  quality  index,  which  is  deriv¬ 
able  from  the  four  steps  above.  The  quality  index  would  provide  NASA 
with  an  estimate  of  the  percentage  of  customer  standards  that  it  is  now  satis¬ 
fying. 

♦  An  implementation  plan  that  prioritizes  the  areas  most  in  need  of  improve¬ 
ment.  The  quality  index  can  be  used  to  show  the  effect  that  closing  each 


4  The  survey  analysis  should  be  conducted  with  a  common  statistical  package  to  fa¬ 
cilitate  analytical  work  and  comparisons.  We  have  designated  the  Statistical  Analysis 
System  for  this  purpose. 


13 


delivery— expectation  gap  would  have  on  NASA's  overall  quality  perform¬ 
ance.  The  customer's  concerns  with  expenditures  also  should  be  compared 
with  quality-improvement  possibilities.  In  this  way,  NASA  can  target  the 
most  fruitful  avenues  for  process  improvement  and  performance  enhance¬ 
ment. 

Initially,  customer  surveys  may  be  difficult  to  plan,  conduct,  and  analyze. 
However,  survey  work  becomes  easier  as  procedures  and  practices  become  more 
routine.  It  is  difficult  to  develop  a  useful  and  unbiased  customer  survey  instru¬ 
ment  for  the  first  time.  It  is  also  difficult  to  conduct  a  survey  properly  in  order  to 
obtain  reasonably  high  response  rates.  Finally,  it  is  difficult  to  devise  a  structure 
for  undertaking  comprehensive  survey  analyses.  As  the  SEUs  continuously  use 
this  guidance,  customer  surveys  will  become  less  difficult. 
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Chapter  4 


Performance  Measurement  System 


Current  Efforts 

The  National  Aeronautics  and  Space  Administration  already  has  a  focus  on 
quality:  continual  improvement.  It  also  has  been  measuring  productivity  for 
many  years,  relating  outputs  to  inputs.  NASA  should  continue  these  efforts  for 
further  improvement  initiatives.  However,  the  Government  Performance  and 
Results  Act  of  1993  (as  well  as  benchmarking)  specifically  requires  that  quality 
and  productivity  be  reflected  in  costs  to  formulate  performance  budgets.  So, 
NASA  will  have  to  take  its  current  performance  improvement  initiatives  one  step 
further  to  meet  the  requirements  of  the  Government  Performance  and  Results 
Act. 


In  this  chapter,  we  define  a  performance  measurement  system  that  ad¬ 
dresses  all  of  NASA's  immediate  and  long-term  performance  initiatives— Execu¬ 
tive  Order  12862,  continual  improvement,  and  the  Government  Performance  and 
Results  Act  of  1993. 


Measurement  Framework 

A  performance  measurement  system  must  provide  comprehensive  signals 
for  an  organization  to  improve  performance.  We  believe  that  a  comprehensive 
performance  measurement  system  has  three  essential  components:  the  ability  to 
enhance  quality,  to  raise  output  capability,  and  to  lower  costs.  Quality  includes 
product  or  service  characteristics  as  well  as  schedule  or  timeliness.  Failure  to 
address  any  of  these  concerns  will  always  result  in  performance  shortfalls.  If  an 
organization  focuses  largely  on  quality,  it  may  very  well  enhance  quality,  but  its 
output  and  costs  could  suffer.  (Mercedes  Benz  could  be  a  good  example  of  an 
organization  that  focused  too  much  on  quality  and  not  enough  on  other  factors.) 
Alternatively,  if  an  organization  focuses  mostly  on  output,  it  may  very  well  raise 
output  capability,  but  its  quality  and  costs  could  suffer.  (The  American  car  in¬ 
dustry  in  the  early  1980s  may  be  a  good  example  of  this  type  of  focus.)  Finally,  if 
an  organization  focuses  largely  on  reducing  costs,  it  may  very  well  become  more 
efficient,  but  its  output  and  quality  might  suffer  (e.g.,  the  Yugo).  Only  when  an 
organization  focuses  simultaneously  on  quality,  output,  and  cost  can  it  truly  es¬ 
tablish  a  high  level  of  overall  performance  and  customer  satisfaction. 

We  use  the  concept  of  performance  unit  cost  to  ensure  that  organizations  fo¬ 
cus  on  all  aspects  of  performance  simultaneously.  Performance  unit  cost  is 
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defined  as  cost  relative  to  outputs  at  defined  quality  levels.  Figure  3  illustrates 
the  process  for  integrating  cost,  output,  and  quality. 

Performance  unit  cost  incorporates  the  best  features  of  Federal,  state,  and  lo¬ 
cal  government  budgeting  efforts.  Measures  of  efficiency  (cost  per  unit  of  out¬ 
put)  have  been  used  extensively  in  the  Federal  government,  especially  with  the 
Department  of  Defense.  At  state  and  local  governments,  measures  of  effective¬ 
ness  or  outcome  (output  adjusted  for  quality  differences)  also  have  been  used  ex¬ 
tensively.  More  recently,  state  and  local  governments  have  combined 
effectiveness  and  efficiency  measures  into  performance  unit  cost  or  cost  per  unit 
of  outcome.  The  Governmental  Accounting  Standards  Board  recently  has  en¬ 
dorsed  this  approach.5 

We  now  define  the  elements  of  performance  unit  cost.  First,  we  discuss  the 
concepts  of  cost,  output,  and  quality.  Then  we  discuss  the  integration  of  those 
concepts  into  a  performance  unit  and  performance  unit  cost. 


Cost  (C) 


Efficiency  (C/0) 


Output  (O) 


Quality  (Q) 


Performance 
unit  cost 
[C/(OxQ)] 


Figure  3. 

Measurement  Integration  Process 


Cost 


Ideally,  process  costs  should  be  inclusive  of  operating  outlays;  appropriate 
allocations  of  general,  administrative,  and  other  fixed  costs;  and  depreciation 
charges  for  use  of  capital  items  (such  as  buildings,  facilities,  and  structures). 
However,  many  government  cost  accounting  systems  cannot  support  this 

5  Proposed  Statement  of  the  Governmental  Accounting  Standards  Board  on  Concepts  Related 
to  Service  Efforts  and  Accomplishing  Reporting,  Governmental  Accounting  Standards  Board, 
September  15, 1993. 
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concept  of  costs  without  substantial  upgrades.  Some  of  the  problems  with  cost 
accounting  systems  include  the  following:  they  cannot  readily  track  variable 
costs  to  specific  processes,  they  do  not  include  procedures  for  depreciating  capi¬ 
tal  items,  and  they  lack  the  capability  to  properly  allocate  fixed  costs  to  proc¬ 
esses.  Recognizing  such  potential  difficulties,  the  Government  Performance  and 
Results  Act  of  1993  does  not  require  full  implementation  of  sophisticated  cost  ac¬ 
counting  and  performance  budgeting  until  the  year  2000.  Also,  to  be  most  effec¬ 
tive,  benchmarking  and  continual  improvement  require  such  cost  accounting 
systems.  NASA  should  ensure  that  its  cost  accounting  and  budgeting  systems 
are  sufficient  for  meeting  these  performance  initiatives. 


Output 

Some  NASA  processes  may  produce  homogeneous  products/ services,  but 
most  are  likely  to  produce  joint  or  multiple  products/  services.  The  measure¬ 
ment  of  homogeneous  outputs  is  straightforward -simply  add  the  number  of 
units  produced  (e.g.,  service  hours)  during  a  given  time  period.  But,  the  meas¬ 
urement  of  diverse  products /  services  is  more  difficult  and  requires  the  use  of  in¬ 
dexes. 

Table  3  illustrates  the  construction  of  output  indexes  when  the  process 
yields  three  different  kinds  of  services.  Suppose  the  different  services  are  S(l), 
S(2),  and  S(3).  Each  could  be  measured  in  terms  of  hours  used;  but,  because  they 
contribute  differently  to  the  total  process  output,  they  must  be  weighted  to  take 
account  of  the  differences.  The  weights,  denoted  by  w(i),  might  be  relative  ex¬ 
penditures  or  revenues  and  are  fixed  over  time  and  sum  to  unity.  The  contribu¬ 
tion  of  each  service  to  the  total  output  is  its  service  level  multiplied  by  its 
respective  weight  [e.g.,  S(l)  x  w(l)],  and  the  summation  of  all  service  contribu¬ 
tions  provides  an  indication  of  total  output. 


Table  3. 

Constructing  Output  Indexes 


Output  service 
(S) 

Importance  weight 
(w) 

Index  contribution 
(Sxw) 

S(1) 

w(1) 

S0)xw(1) 

S(2) 

w(2) 

S(2)xw(2) 

S(3) 

w(3) 

S(3)xw(3) 

Total 

1 

Index  value  j 

Quality 


We  now  need  to  define  quality  factors  for  NASA's  products/ services  and  to 
express  NASA's  outputs  in  quality-equivalent  terms.  NASA's  products  and 
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services  normally  are  expected  to  involve  multiple  quality  factors  —  such  as  suit¬ 
ability,  reliability,  and  usability  —  so  the  same  basic  index  approach  used  to 
summarize  outputs  also  has  application  to  the  summation  of  quality.  However, 
two  major  differences  in  constructing  quality  indexes  exist. 

First,  unlike  those  in  the  output  index,  the  weights  in  quality  indexes  are  de¬ 
rived  from  customer  satisfaction  surveys  (as  required  by  Executive  Order  12862) 
and  not  from  NASA's  revenue  or  expenditure  data.  In  the  survey,  customers 
would  rate  the  importance  of  each  quality  factor,  and  those  ratings  would  be 
used  to  construct  the  quality  weights  in  the  index. 

Second,  the  performance-of-quality  factor  is  measured  in  percentage 
terms  —  rather  than  in  actual  values  as  in  the  case  of  output  measurement  —  to 
determine  how  the  actual  quality  levels  compare  to  customer  standards. 
Moreover,  benchmarking  factors  could  also  be  used  as  standards  in  the  quality 
index,  but  they  would  have  to  be  at  least  as  high  as  the  customer's  standards. 
We  believe,  however,  that  benchmarking  should  be  used  only  after  using  cus¬ 
tomer  standards  for  a  period  of  time. 

Equation  2  shows  the  general  form  of  the  quality  index  with  "n"  quality  fac¬ 
tors: 


w(n)x 


actual  qualityin ) 
standard  quality(n)  * 


[Eq.2] 


The  quality  index  then  measures  the  degree  to  which  the  quality  standards 
are  met.  For  example,  an  80  percent  quality  index  score  would  mean  that,  on  the 
average,  the  process  is  achieving  80  percent  of  the  quality  standards  desired  by 
the  customers.  The  maximum  quality  index  score  is  100  percent,  and  the  mini¬ 
mum  is  0  percent. 

Equation  2  is  also  a  useful  tool  for  determining  the  quality  factors  that  have 
the  greatest  need  for  improvement.  If  two  quality  factors  achieve  only  50  percent 
of  their  standard  levels,  for  example,  then  the  one  with  the  greater  customer 
weight  should  receive  the  higher  priority  for  making  improvements.  In  contrast, 
a  quality  factor  with  a  90  percent  achievement  level  relative  to  its  standard  (and  a 
relatively  low  customer  weight)  should  receive  a  lower  priority  for  implement¬ 
ing  improvements. 


Performance  Unit 

Performance  units  are  outputs  standardized  at  perfect  quality  levels.  Such 
standardization  is  needed  to  measure  performance  accurately  and  to  encourage 
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high-quality  performance.  Table  4  illustrates  three  different  cases.  Case  "a" 
shows  that  1,200  units  of  output  were  produced  at  an  average  of  50  percent  of  the 
standards.  Its  performance  units  totaled  600  (1,200  x  0.50).  In  case  "b,"  perform¬ 
ance  is  higher,  at  960  units  (1,200  x  0.80),  reflecting  a  higher  level  of  quality.  Case 
"c"  shows  the  lowest  number  of  performance  units,  560  (1,400  x  0.40),  because  of 
a  significantly  lower  quality  level.  These  three  cases  illustrate  that  accurate  per¬ 
formance  measurement  must  standardize  for  quality  differences  and  that  organi¬ 
zations  could  perform  poorly  if  they  do  not  embrace  such  comprehensive 
measures. 

Table  4. 

Performance  Units 


Case 

Output 

(O) 

Quality  index 
(Qi) 

Performance  unit 
(OxQI) 

a 

U  - 1 

0.50 

600 

b 

1 

0.80 

960 

c 

1,400 

0.40 

560 

Performance  Unit  Cost 

The  starting  point  for  defining  performance  unit  cost  is  to  establish  the  rela¬ 
tionship  between  total  costs,  on  die  one  hand,  and  output  and  quality,  on  the 
other.  Economists  call  such  a  relationship  a  "total  cost  function";  its  properties 
can  be  derived  by  either  regression  analysis  or  engineering  value  analysis. 
Skilled  professionals  would  be  needed  to  undertake  such  technical  analyses.6 
Equation  3  represents  a  total  cost  function. 

Total  cost  -/{output,  quality).  [Eq.  3] 

It  indicates  that  costs  vary  with  both  output  and  quality  and  that  such  a  relation¬ 
ship  could  be  either  linear  or  nonlinear,  depending  upon  the  particular  circum¬ 
stances. 

Performance  unit  cost  (or  PUC)  simply  expresses  the  total  cost  relative  to 
performance  units.  Dividing  both  sides  of  Equation  3  by  the  performance  units 
(output  x  quality),  we  can  calculate  PUC,  as  Equation  4  shows: 

PUC  -/{output,  quality)! petformance  units.  [Eq.  4] 


6  See  Jan  Kmenta,  Elements  of  Econometrics,  New  York:  Macmillan  Publishing  Co.,  Inc., 
1971,  for  a  good  treatment  of  regression  analysis.  Also,  see  David  K.  Burt,  et  al..  Zero 
Base  Pricing:  Achieving  World  Class  Competitiveness  Through  Reduced  All-in  Costs,  Chicago: 
Probus  Publishing  Company,  1990,  for  a  thorough  explanation  of  engineering  value 
analysis. 
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Note  that  PUC  is  not  a  constant  but  varies  with  output  and  quality.  Many 
budgeting  systems  assume  that  PUC  is  constant.  Such  an  assumption,  however, 
could  lead  to  incorrect  managerial  decisions,  especially  when  related  to  consoli¬ 
dations  and  realignments.  Organizations  should  use  a  constant  PUC  only  as  a 
very  crude  first  approximation. 

Figure  4  shows  the  PUC  for  a  process  in  which  the  total  cost  function  is  lin¬ 
ear.  In  this  situation,  average  variable  costs  and  marginal  costs  are  constant. 
Thus,  PUC  falls  as  fixed  costs  are  spread  over  more  and  more  performance  units 
(such  as  at  point  "b"),  and  it  approaches  a  constant  (the  marginal  cost)  as  the 
number  of  performance  units  becomes  large  (such  as  at  point  "a").7 


Unit  Performance 

Figure  4. 

Effect  of  Linear  Cost  Function  on  PUC 


The  PUC  curve  can  be  used  for  various  managerial  purposes.  For  instance, 
it  could  be  used  to  summarize  organizational  progress  for  making  improve¬ 
ments.  Suppose  an  organization  initially  operates  at  point  "b,"  signifying  a  rela¬ 
tively  high  PUC.  That  organization  might  pursue  cost  reduction  and/or  quality 
initiatives  to  improve  its  performance.  If  it  simply  reduces  its  costs  for  that  level 
of  performance  units  (such  as  at  point  "c"),  then  its  PUC  would  decline 
from  PUC  (b)  to  PUC  (c).  If  it  focuses  on  quality  enhancement  (such  as  at  point 
"a"),  then  its  PUC  also  would  decline,  but  less  significantly— from  PUC  (b)  to 


7  An  example  of  a  linear  cost  function  might  be  as  follows:  Total  cost  =  a  +  b  (output  x 
quality  index).  In  this  situation,  marginal  cost  is  equal  to  "b"  and  is  constant. 
PUC  =  a/ (output  x  quality  index)  +  b,  with  the  first  term  reflecting  average  fixed  costs 
(which  vary  inversely  with  output  and  quality)  and  the  last  term  reflecting  the  constant 
average  variable  or  marginal  costs. 
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PUC  (a).  Either  enhancing  quality  or  reducing  costs  may  be  appropriate,  de¬ 
pending  upon  customer  feedback. 

The  PUC  curve  could  also  be  used  to  target  improvement  initiatives.  If  cus¬ 
tomers  indicate  that  costs  are  the  primary  problem,  then  an  organization  s  strat¬ 
egy  might  be  to  undertake  cost-reduction  efforts  (such  as  plans  to  move  from 
point  "b"  to  "c"  in  Figure  4).  However,  if  customers  indicate  that  quality  is  their 
major  problem,  then  the  appropriate  strategy  might  be  to  undertake  quality  im¬ 
provement  efforts  (such  as  plans  to  move  from  point  b  to  a  ). 

Finally,  the  PUC  curve  could  be  used  to  develop  budgets.  As  an  example, 
the  budget  required  to  operate  at  performance  units  of  UP  (a)  would  be  PUC  (a) 
x  UP  (a);  or,  the  budget  to  operate  at  UP  (b)  performance  units  would  be  PUC 
(c)x  UP  (b)  after  cost-reduction  efforts. 

Appendix  B  applies  the  performance  unit  cost  system  to  the  wind-tunnel 
process  at  Ames  Research  Center.  It  illustrates  what  is  involved  in  fully  measur¬ 
ing  performance  to  meet  the  benchmarking  requirements  of  Executive 
Order  12862,  the  goals  of  NASA's  continual  improvement  program,  and  the 
challenges  of  the  Government  Performance  and  Results  Act  of  1993. 
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Appendix  A 


Executive  Order  12862 
of  September  11, 1993, 

Setting  Customer  Service  Standards 


Putting  people  first  means  ensuring  that  the  Federal  Government  provides 
the  highest  quality  service  possible  to  the  American  people.  Public  officials  must 
embark  upon  a  revolution  within  the  Federal  Government  to  change  the  way  it 
does  business.  This  will  require  continual  reform  of  the  executive  branch's  man¬ 
agement  practices  and  operations  to  provide  service  to  the  public  that  matches  or 
exceeds  the  best  service  available  in  die  private  sector. 

NOW,  THEREFORE,  to  establish  and  implement  customer  service  standards 
to  guide  the  operations  of  the  executive  branch,  and  by  the  authority  vested  in 
me  as  President  by  the  Constitution  and  the  laws  of  the  United  States,  it  is  hereby 
ordered: 

Section  1.  Customer  Service  Standards.  In  order  to  carry  out  the  principles  of 
the  National  Performance  Review,  the  Federal  Government  must  be  customer- 
driven.  The  standard  of  quality  for  services  provided  to  the  public  shall  be:  Cus¬ 
tomer  service  equal  to  the  best  in  business.  For  the  purposes  of  this  order,  cus¬ 
tomer"  shall  mean  an  individual  or  entity  who  is  directly  served  by  a  department 
or  agency.  "Best  in  business"  shall  mean  the  highest  quality  of  service  delivered 
to  customers  by  private  organizations  providing  a  comparable  or  analogous 
service. 

All  executive  departments  and  agencies  (hereinafter  referred  to  collectively  as 
"agency"  or  "agencies")  that  provide  significant  services  directly  to  the  public 
shall  provide  those  services  in  a  manner  that  seeks  to  meet  the  customer  service 
standard  established  herein  and  shall  take  the  following  actions: 

(a)  identify  the  customers  who  are,  or  should  be,  served  by  the  agency; 

(b)  survey  customers  to  determine  the  kind  and  quality  of  services  they  want 
and  their  level  of  satisfaction  with  existing  services; 

(c)  post  service  standards  and  measure  results  against  them; 

(d)  benchmark  customer  service  performance  against  the  best  in  business; 

(e)  survey  front-line  employees  on  barriers  to,  and  ideas  for,  matching  the  best 
in  business; 
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(f)  provide  customers  with  choices  in  both  the  sources  of  service  and  the  means 
of  delivery; 

(g)  makes  information,  services,  and  complaint  systems  easily  accessible;  and 

(h)  provide  means  to  address  customer  complaints. 

Section  2.  Report  on  Customer  Service  Surveys.  By  March  8, 1994,  each  agency 
subject  to  this  order  shall  report  on  its  customer  surveys  to  the  President.  As  in¬ 
formation  about  customer  satisfaction  becomes  available,  each  agency  shall  use 
that  information  in  judging  the  performance  of  agency  management  and  in  mak¬ 
ing  resource  allocations. 

Section  3.  Customer  Service  Plans.  By  September  8, 1994,  each  agency  subject  to 
this  order  shall  publish  a  customer  service  plan  that  can  be  readily  understood 
by  its  customers.  The  plan  shall  include  customer  service  standards  and  describe 
future  plans  for  customer  surveys.  It  also  shall  identify  the  private-  and  public- 
sector  standards  that  the  agency  used  to  benchmark  its  performance  against  the 
best  in  business.  In  connection  with  the  plan,  each  agency  is  encouraged  to  pro¬ 
vide  training  resources  for  programs  needed  by  employees  who  directly  serve 
customers  and  by  managers  making  use  of  customer  survey  information  to  pro¬ 
mote  the  principles  and  objectives  contained  herein. 

Section  4.  Independent  Agencies.  Independent  agencies  are  requested  to  adhere 
to  this  order. 

Section  5.  Judicial  Review.  This  order  is  for  the  internal  management  of  the  ex¬ 
ecutive  branch  and  does  not  create  any  right  or  benefit,  substantive  or  proce¬ 
dural,  enforceable  by  a  party  against  the  United  States,  its  agencies  or 
instrumentalities,  its  officers  or  employees,  or  any  other  person. 

WILLIAM  J.  CLINTON 


THE  WHITE  HOUSE 
September  11, 1993. 


Appendix  B 

Wind-Tunnel  Case 


Using  our  definition  of  processes,  the  National  Aeronautics  and  Space  Ad¬ 
ministration's  (NASA's)  Ames  Research  Center  (ARC)  wind-tunnel  operations 
could  be  classified  as  a  single  process  for  measuring  performance  and  making 
improvements.  It  is  our  understanding  that  ARC  operates  three  different  types 
of  wind  tunnels:  a  7-by-10-foot  high-speed  tunnel,  a  40-by-80-foot  moderate- 
speed  tunnel,  and  an  80-by-120-foot  low-speed  tunnel.  Each  of  those  tunnels 
supports  different  customers.  Also,  ARC  schedules  wind-tunnel  use  in  order  to 
minimize  power,  maintenance,  and  repair  costs.  Finally,  the  ARC  wind  tunnels 
further  support  NASA's  own  operations  (e.g.,  in-flight  orbiter  preparations)  and 
industry  (e.g.,  in  helicopter  research). 

In  illustrating  our  performance  measurement  system  with  the  ARC  wind- 
tunnel  operation  as  a  single  process,  we  need  to  first  define  and  formulate  the 
elements  of  performance  units  for  the  single  wind-tunnel  process:  cost,  output, 
and  quality.  Then,  we  need  to  formulate  the  total  cost  and  (performance  unit 
cost  1  PUC)  curve,  as  well  as  illustrate  their  application  to  management. 


Cost 


The  total  costs  (TC)  for  ARC's  wind-tunnel  process  may  be  broken  down 
into  two  components:  operating  costs  and  depreciation  charges.  Operating  costs 
(OQ  consist  of  power,  maintenance,  repairs,  and  other  variable  costs,  as  well  as 
some  administrative  and  fixed  costs  that  are  associated  with  the  wind-tunnel  op¬ 
eration.  Depreciation  charges  (DC)  reflect  the  amortization  of  expenditures  for 
the  wind-tunnel  facilities,  equipment,  and  other  capital  items.  Equation  B-l 
shows  the  straightforward  calculation  of  TC,  in  constant  dollars.  It  reflects  the 
real  resources  necessary  to  provide  wind-tunnel  services  at  certain  levels  of  qual¬ 
ity. 


TC=  OC+DC. 


[Eq.  B-l] 


Output 


The  wind-tunnel  outputs  consist  of  low-,  medium-,  and  high-speed  wind- 
tunnel  services.  Each  of  those  service  outputs  could  be  measured  in  terms  of  us¬ 
age  hours— low-(l),  medium-(m),  or  high-(h)  speed  hours,  hi  order  to  combine 
these  outputs  into  a  composite,  we  need  to  weight  them  to  reflect  their  relative 
contributions  to  the  total  ARC  wind-tunnel  service.  We  believe  that  relative  cost 
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weights  would  best  serve  this  purpose.  Equation  B-2  shows  the  calculation  of 
the  output  index  (OI)  for  the  ARC  wind-tunnel  process. 

01  =  w(  1 )  x  / + w( 2)  xm  +  w(3)xh.  [Eq.  B-2] 


Quality 


The  wind-tunnel  services  are  very  likely  to  vary  in  quality  in  a  number  of  re¬ 
spects.  We  have  chosen  three  quality  dimensions  to  illustrate  this  point:  suitabil¬ 
ity,  usability,  and  reliability.  Suitability  relates  to  the  adequacy  of  the  ARC  wind 
tunnels  offered  for  meeting  the  testing  requirements  of  its  customers.  Usability 
refers  to  whether  the  wind-tunnel  service  schedules  are  reasonable  for  meeting 
the  needs  of  both  NASA  and  industry  customers.  Reliability  indicates  whether 
the  wind  tunnel  operation  performs  well  and  without  interruption.  Each  of 
these  quality  dimensions  is  expressed  as  a  percentage  of  actual  performance  as 
compared  to  the  standard  measures.  We  then  need  to  combine  these  quality 
measures  into  an  overall  quality  index  (QI)  using  Equation  B-3. 

QI=i(l)x  (%  suit.)  +  i( 2) x  (%  use.)  +  x( 3)  x  (%  rel.).  [Eq.  B-3] 

The  quality  factor  weights  in  the  index  -i(l),  i(2),  and  i(3)  should  reflect  the 
relative  importance  that  industry  and  NASA  customers  attach  to  them.  As  ex¬ 
plained  in  Chapter  3,  a  customer  survey  could  use  a  l-to-7  scoring  scale  to  deter¬ 
mine  customer  standards  and  satisfaction  for  meeting  those  standards. 


Performance  Unit  Cost 

To  obtain  performance  unit  cost  for  the  ARC  wind-tunnel  process,  we  need 
to  follow  three  sequential  steps: 

1.  Define  unit  performance  (UP)  using  the  formula  UP  =  OI  x  QI,  where  OI  is 
computed  using  Equation  B-2  and  QI  is  computed  using  Equation  B-3. 

2.  Establish  the  relationship  between  total  cost  and  the  quality  and  output  in¬ 
dexes.  Both  linear  and  nonlinear  relationships  should  be  considered;  they 
can  be  established  by  using  either  regression  analysis  or  engineering  value 
analysis. 

3.  Calculate  performance  unit  cost  by  dividing  the  UP  measure  from  step  1 
into  the  cost  equation  established  in  step  2. 
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Management  Uses 

With  the  quality  index,  ARC  can  then  determine  where  it  needs  to  make  im¬ 
provements  by  expressing  the  quality  index  (Equation  B-3)  in  change  or  com¬ 
plete  difference  form  as  in  Equation  B-4: 

d(QI)  =  i(  1)  xd(%  suit.)  gap  +  i( 2)  x  d(%  use.)  gap  +  /( 3)  x  d(%  rel.)gap.  [Eq.  B-4] 

The  gap  for  each  quality  factor  indicates  the  remaining  difference  between 
the  customer's  standard  and  actual  satisfaction  level.  For  example,  if  a  customer 
indicates  that  60  percent  of  the  standard  level  is  met  for  suitability,  the  remaining 
actual-standard  gap  is  40  percent.  If  the  weight  for  suitability  is  0.60,  then  im¬ 
provements  in  suitability  can  contribute  a  maximum  of  24  percent  to  overall 
wind-tunnel  process  quality  (40  percent  gap  x  0.60  weight  in  quality  index). 
ARC  could  also  determine  the  relative  contributions  of  the  other  quality  factors 
to  the  overall  potential  improvement  in  ARC  wind-tunnel  process  quality. 
Armed  with  this  information,  ARC  then  could  prioritize  its  improvement  efforts 
according  to  their  relative  contributions  to  changes  in  overall  quality. 

ARC  could  use  this  same  performance  measurement  system  for  other  mana¬ 
gerial  purposes.  Through  the  customer  survey,  it  could  determine  the  impor¬ 
tance  of  wind-tunnel  expenditures  to  the  users  and  then  compare  that  result  to 
potential  quality  initiatives.  If  the  customers  consider  expenditures  more  impor¬ 
tant  than  quality,  ARC  should  focus  its  process  improvements  on  reducing  costs. 
Finally,  ARC'S  overall  progress  on  wind-tunnel  process  improvements  and 
budgeting  requirements  could  be  represented  by  the  performance  unit  cost 
curves. 


